feat: Added in-memory storage for testing purposes #59

Harshdev098 · 2025-10-12T08:40:04Z

Have added in_memory store for testing purpose.
We can edit config file to use specific store either postgresql or memory

ldk-reviews-bot · 2025-10-12T08:40:06Z

👋 Thanks for assigning @tnull as a reviewer!
I'll wait for their review and will help manage the review process.
Once they submit their review, I'll check if a second reviewer would be helpful.

Harshdev098 · 2025-10-12T10:19:04Z

Hey @tnull @tankyleo Can you please review it

tnull

Thanks for looking into this!

Generally goes into the right direction, but we def. need to avoid re-allocating everything on every operation.

rust/Cargo.lock

rust/README.md

rust/impls/src/lib.rs

rust/impls/src/memory_store.rs

Harshdev098 · 2025-10-14T06:17:50Z

@tnull Have done the required changes

tnull

Looks much better, but I think we still need to handle global_version properly, even if we're currently not using it client-side.

rust/impls/src/in_memory_store.rs

ldk-reviews-bot · 2025-10-14T08:50:52Z

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-10-16T08:51:33Z

🔔 2nd Reminder

Hey @tnull @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-10-16T16:44:43Z

🔔 1st Reminder

Hey @tnull @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tnull

One comment, will take another look once @tankyleo also had a chance to do a review round here.

rust/impls/src/in_memory_store.rs

Harshdev098 · 2025-10-17T09:15:56Z

@tankyleo Can you please review it!

ldk-reviews-bot · 2025-10-18T08:52:14Z

🔔 3rd Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-10-20T08:52:50Z

🔔 4th Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-10-22T08:53:27Z

🔔 5th Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tankyleo

Sorry for the delay !

rust/impls/src/in_memory_store.rs

tankyleo · 2025-10-22T23:59:11Z

rust/impls/src/in_memory_store.rs

+					version: record.version,
+				}),
+			})
+		} else if request.key == GLOBAL_VERSION_KEY {


Looks like by the time we are here, we know the GLOBAL_VERSION_KEY does not have a value, otherwise guard.get would have returned Some previously. We can just return the GetObjectResponse below directly with version: 0.

@Harshdev098 double checking things here, we still have this second branch the same as before ?

rust/impls/src/in_memory_store.rs

tankyleo · 2025-10-23T00:05:55Z

rust/impls/src/in_memory_store.rs

+		let key_prefix = request.key_prefix.unwrap_or_default();
+		let page_token = request.page_token.unwrap_or_default();


If you don't mind, let's use the more explicit unwrap_or here like we have just below. unwrap_or_default is a little hard to read.

@Harshdev098 help me understand we still have the unwrap_or_default here ? If you would prefer unwrap_or_default let me know.

Ahh, actually I was debugging the issue of CI and have forgot to use unwrap_or, will update it!

rust/impls/src/in_memory_store.rs

rust/impls/src/migrations.rs

Harshdev098 · 2025-10-26T01:53:09Z

@tankyleo Have done with the required changes! Can you please review it

tnull

When testing integration with LDK Node locally I found that the tests are currently failing. I now opened #62 to add LDK Node integration tests to our CI here. It would be great if that could land first, and we could also add a CI job for the in-memory store as part of this PR then, ensuring the implementation actually works as expected.

ldk-reviews-bot · 2025-10-28T01:53:31Z

🔔 1st Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-10-30T01:54:18Z

🔔 2nd Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tnull · 2025-10-31T12:09:51Z

@Harshdev098 Please rebase now that #62 landed to make use of the new CI checks here.

tnull

Please also make sure that the LDK Node CI is run against the in-memory store.

AFAICT this is still missing the part that runs the LDK Node integration test against the in-memory store to check functionality in a realistic setting?

ldk-reviews-bot · 2025-11-05T01:55:32Z

🔔 5th Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tnull

Right, and now you're hitting the errors that I originally found when running LDK Node against the in-memory store. Very likely, they are due to the in-memory implementation behaving in an unexpected manner. We need to fix this before this PR can get merged. Please take a look, but let me know if you end up getting stuck or need some assistance figuring this out.

ldk-reviews-bot · 2025-11-08T00:01:03Z

🔔 6th Reminder

Hey @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

Harshdev098 · 2025-11-08T09:36:32Z

Hey @tnull Have updated the code and the unit test are passing against the ldk node tests but didn't understand what is the cause of integration failures

Harshdev098 · 2025-11-09T03:53:12Z

Hey @tnull @tankyleo Have tested it locally, the test are working correctly now, don't know why its stuck in CI! Can you please review it

ldk-reviews-bot · 2025-11-10T00:01:47Z

🔔 7th Reminder

Hey @tnull @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-11-11T03:54:09Z

🔔 1st Reminder

Hey @tnull @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

ldk-reviews-bot · 2025-11-12T00:02:20Z

🔔 8th Reminder

Hey @tnull @tankyleo! This PR has been waiting for your review.
Please take a look when you have a chance. If you're unable to review, please let us know so we can find another reviewer.

tankyleo

Sorry for the delay again I have cleared all the other priorities, this is now top priority :) Here are some comments, mostly on the types we pass to the different functions.

Will continue review tomorrow 100% ! Thank you again.

tankyleo · 2025-11-12T02:35:23Z

rust/impls/src/postgres_store.rs

-			ErrorKind::Other,
-			format!("Failed to drop database {}: {}", db_name, e),
-		)
+		Error::new(ErrorKind::Other, format!("Failed to drop database {}: {}", db_name, e))


Sorry the formats in this file were done in a CI-specific PR we merged, go ahead and rebase and drop them thank you :)

@Harshdev098 sorry i take it back, don't rebase to a new commit just yet :) if you can just drop these changes, keep the same base commit on main, will rebase to a newer base commit as necessary thank you

Ohh sorry @tankyleo , but I have done it already, my branch is now up to date with main

tankyleo · 2025-11-12T02:41:46Z

rust/impls/src/in_memory_store.rs

+		let mut vss_put_records: Vec<VssDbRecord> = request
+			.transaction_items
+			.into_iter()
+			.map(|kv| build_vss_record(user_token.clone(), store_id.clone(), kv))
+			.collect();
+
+		let vss_delete_records: Vec<VssDbRecord> = request
+			.delete_items
+			.into_iter()
+			.map(|kv| build_vss_record(user_token.clone(), store_id.clone(), kv))
+			.collect();


Let's not build all these VssDbRecords when we still haven't validated the API input. Instead, let's pass the KeyValue's by reference to the validator functions (ie just read the input to validate it, no cloning / allocations).

Then once we've finished validation, we can build the VssDbRecord and insert into the db as needed.

My fn put function looks like this, feel free to grab it and make the changes you want. I create the VssDbRecord inside the execute_put_object function.

async fn put( &self, user_token: String, request: PutObjectRequest, ) -> Result<PutObjectResponse, VssError> { if request.transaction_items.len() + request.delete_items.len() > MAX_PUT_REQUEST_ITEM_COUNT { return Err(VssError::InvalidRequestError(format!( "Number of write items per request should be less than equal to {}", MAX_PUT_REQUEST_ITEM_COUNT ))); } let store_id = request.store_id.clone(); let mut guard = self.store.write().await; if let Some(version) = request.global_version { validate_put_operation(&guard, &user_token, &store_id, &KeyValue { key: GLOBAL_VERSION_KEY.to_string(), value: Bytes::new(), version, })?; } for key_value in &request.transaction_items { validate_put_operation(&guard, &user_token, &store_id, key_value)?; } for key_value in &request.delete_items { validate_delete_operation(&guard, &user_token, &store_id, key_value)?; } for key_value in request.transaction_items { execute_put_object(&mut guard, &user_token, &store_id, key_value); } for key_value in &request.delete_items { execute_delete_object(&mut guard, &user_token, &store_id, key_value); } if let Some(version) = request.global_version { execute_put_object(&mut guard, &user_token, &store_id, KeyValue { key: GLOBAL_VERSION_KEY.to_string(), value: Bytes::new(), version, }); } Ok(PutObjectResponse {}) }

tankyleo · 2025-11-12T02:43:19Z

rust/impls/src/in_memory_store.rs

+		let vss_delete_records: Vec<VssDbRecord> = request
+			.delete_items
+			.into_iter()
+			.map(|kv| build_vss_record(user_token.clone(), store_id.clone(), kv))
+			.collect();


We also do not need to build VssDbRecords for the delete_items, so let's delete those.

tankyleo · 2025-11-12T02:46:19Z

rust/impls/src/in_memory_store.rs

+fn build_vss_record(user_token: String, store_id: String, kv: KeyValue) -> VssDbRecord {
+	let now = Utc::now();
+	VssDbRecord {
+		user_token,
+		store_id,
+		key: kv.key,
+		value: kv.value.to_vec(),
+		version: kv.version,
+		created_at: now,
+		last_updated_at: now,
+	}
+}


Once we only build the VssDbRecord when we are about to insert into the database, we can delete this function here.

tankyleo · 2025-11-12T02:47:19Z

rust/impls/src/in_memory_store.rs

+
+// Validation functions - check for if operations can succeed without modifying data
+fn validate_put_operation(
+	store: &HashMap<String, VssDbRecord>, user_token: &str, store_id: &str, record: &VssDbRecord,


As I described earlier, let's pass a key_value: &KeyValue here not a record: &VssDbRecord

tankyleo · 2025-11-12T02:50:41Z

rust/impls/src/in_memory_store.rs

+
+fn execute_non_conditional_upsert(
+	store: &mut HashMap<String, VssDbRecord>, user_token: &str, store_id: &str,
+	record: &VssDbRecord,


We are inserting the KeyValue into the database here, so let's pass a key_value: KeyValue here instead of the record: &VssDbRecord

tankyleo · 2025-11-12T02:51:46Z

rust/impls/src/in_memory_store.rs

+
+fn execute_conditional_insert(
+	store: &mut HashMap<String, VssDbRecord>, user_token: &str, store_id: &str,
+	record: &VssDbRecord,


Same reasoning as above, let's pass a key_value: KeyValue here

tankyleo · 2025-11-12T02:52:29Z

rust/impls/src/in_memory_store.rs

+
+fn execute_conditional_update(
+	store: &mut HashMap<String, VssDbRecord>, user_token: &str, store_id: &str,
+	record: &VssDbRecord,


Same here, a key_value: KeyValue fits better since we are only modifying an existing VssDbRecord, not creating a new one.

tankyleo · 2025-11-12T02:55:00Z

rust/impls/src/in_memory_store.rs

+
+fn execute_non_conditional_delete(
+	store: &mut HashMap<String, VssDbRecord>, user_token: &str, store_id: &str,
+	record: &VssDbRecord,


Here and below, we only need to pass a key_value: &KeyValue, as we are only reading the key, we do not need a &VssDbRecord

tankyleo · 2025-11-12T02:57:19Z

rust/impls/src/in_memory_store.rs

+
+fn execute_delete_object(
+	store: &mut HashMap<String, VssDbRecord>, user_token: &str, store_id: &str,
+	record: &VssDbRecord,


Also here let's just pass a key_value: &KeyValue, we do not need a record: &VssDbRecord

Harshdev098 · 2025-11-09T10:25:38Z

rust/impls/src/in_memory_store.rs

+		Ok(DeleteObjectResponse {})
+	}
+
+	async fn list_key_versions(


There was a deadlock here earlier have resolved that! now its running locally

Harshdev098 · 2025-11-12T02:55:55Z

rust/impls/src/in_memory_store.rs

+		let key_prefix = request.key_prefix.unwrap_or_default();
+		let page_token = request.page_token.unwrap_or_default();


Ahh, actually I was debugging the issue of CI and have forgot to use unwrap_or, will update it!

Harshdev098 · 2025-11-12T04:50:07Z

rust/impls/src/postgres_store.rs

-			ErrorKind::Other,
-			format!("Failed to drop database {}: {}", db_name, e),
-		)
+		Error::new(ErrorKind::Other, format!("Failed to drop database {}: {}", db_name, e))


Ohh sorry @tankyleo , but I have done it already, my branch is now up to date with main

Harshdev098 · 2025-11-12T05:15:57Z

@tankyleo Have updated the code

ldk-reviews-bot requested a review from tankyleo October 12, 2025 08:50

tnull self-requested a review October 13, 2025 07:10

tnull reviewed Oct 13, 2025

View reviewed changes

Harshdev098 force-pushed the memory_store branch 2 times, most recently from 4980a75 to 25d57e3 Compare October 14, 2025 06:16

Harshdev098 requested a review from tnull October 14, 2025 06:17

tnull reviewed Oct 14, 2025

View reviewed changes

rust/impls/src/in_memory_store.rs Show resolved Hide resolved

Harshdev098 force-pushed the memory_store branch from 25d57e3 to 9012e95 Compare October 14, 2025 12:24

Harshdev098 requested a review from tnull October 14, 2025 16:44

tnull reviewed Oct 17, 2025

View reviewed changes

rust/impls/src/in_memory_store.rs Outdated Show resolved Hide resolved

Harshdev098 force-pushed the memory_store branch from 9012e95 to 3b434d0 Compare October 17, 2025 09:14

tankyleo reviewed Oct 23, 2025

View reviewed changes

Harshdev098 force-pushed the memory_store branch from 3b434d0 to e0c31bb Compare October 26, 2025 01:51

Harshdev098 requested a review from tankyleo October 26, 2025 01:53

Harshdev098 force-pushed the memory_store branch from 8003119 to 5898609 Compare October 26, 2025 02:18

tnull reviewed Oct 27, 2025

View reviewed changes

Harshdev098 requested a review from tnull November 4, 2025 03:45

tnull reviewed Nov 4, 2025

View reviewed changes

Harshdev098 force-pushed the memory_store branch from 81ca808 to 1f18bd8 Compare November 4, 2025 11:41

Harshdev098 force-pushed the memory_store branch 2 times, most recently from 84cfed6 to 939559f Compare November 5, 2025 03:34

tnull reviewed Nov 5, 2025

View reviewed changes

Harshdev098 force-pushed the memory_store branch 4 times, most recently from 51a791d to 0168750 Compare November 8, 2025 05:48

Harshdev098 force-pushed the memory_store branch 3 times, most recently from 92bd1e3 to bd3eca4 Compare November 9, 2025 03:41

Harshdev098 requested a review from tnull November 9, 2025 03:53

Harshdev098 force-pushed the memory_store branch 2 times, most recently from 9ec0e79 to 45fecd1 Compare November 9, 2025 10:18

tankyleo reviewed Nov 12, 2025

View reviewed changes

Harshdev098 added 3 commits November 12, 2025 09:26

chore: formatted and pub(crate) visibility

0bf37eb

feat: Added in-memory storage for testing purposes

02a1b4e

feat: Added CI for in_memory_testing

c2a140e

Harshdev098 commented Nov 12, 2025

View reviewed changes

Harshdev098 force-pushed the memory_store branch from 45fecd1 to c2a140e Compare November 12, 2025 05:10

		let key_prefix = request.key_prefix.unwrap_or_default();
		let page_token = request.page_token.unwrap_or_default();

feat: Added in-memory storage for testing purposes #59

Are you sure you want to change the base?

feat: Added in-memory storage for testing purposes #59

Uh oh!

Conversation

Harshdev098 commented Oct 12, 2025

Uh oh!

ldk-reviews-bot commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Harshdev098 commented Oct 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Harshdev098 commented Oct 14, 2025

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

ldk-reviews-bot commented Oct 14, 2025

Uh oh!

ldk-reviews-bot commented Oct 16, 2025

Uh oh!

ldk-reviews-bot commented Oct 16, 2025

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Harshdev098 commented Oct 17, 2025

Uh oh!

ldk-reviews-bot commented Oct 18, 2025

Uh oh!

ldk-reviews-bot commented Oct 20, 2025

Uh oh!

ldk-reviews-bot commented Oct 22, 2025

Uh oh!

tankyleo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Harshdev098 commented Oct 26, 2025

Uh oh!

tnull left a comment

Choose a reason for hiding this comment

Uh oh!

ldk-reviews-bot commented Oct 28, 2025

Uh oh!

ldk-reviews-bot commented Oct 30, 2025

Uh oh!

tnull commented Oct 31, 2025

ldk-reviews-bot commented Oct 12, 2025 •

edited

Loading

Harshdev098 commented Oct 12, 2025 •

edited

Loading